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OPERATIONSRESEARCH AS -A MSXAPHOR FOR EVALUATIOH 

• ' % ♦ 

A favorite colleague, given to mlschieC-tnaking, has described evaluation 
^ as »'50 percent measurement and 50 percent nush*** Iri\that part which is 
not measurement^ he clalmSf there Is little Intelleccvial coherence, and no 
sctentVflc respectability. Regarding evaluation its prqsent 
state, ont finds, it hard to disagree t* ' (Page, 1975, 1978). 
However, the argument of this thaptei is that the ••mush" may. have made us' 
*iling,but not terminally HI* Indeed, there are objective principles, 
of the - - sort'vhich have lei to the success of oeaaur^nt and of oost 
\ scientific activity, which can be applied through the field of evaluation, 
and can glvc^ic an Integrity and legitimacy which aye only, fantasies today. 
These saving principlfea are those of Operations Research (oR), , 



Vb^ti^iantly unfolding discipline less than «J years old.", Probably not one 
porceni of today's cvaluator8,,even those wis,th extensive statistical , 
knowled^ Icno*?sOR wltix. ?nV <iepth. Yet this 41sclpllne;addretfses ' 

■ 

• exactly those problems closest to our fcearts*, those of daclslon-toakinfi. 
. Indeed, operations research tg -the science of evaluation. 

This artl6le cin only be a'brlef Overview of OR, atyJ how It relates'to* 
. evaluation. First, we shall demonstrate that flducatl^nal evaluation (EE) 
t» indeed' .targeted on decision-making. Second, we shall consider how 
one iwbiield of:OR, decision analysis, Lkes explicit and, tractable v 
the". eleaents o£ our concern's. Third, ve shall illftstrate some of the 
other subfields of OR and pesslbU applicati.ons in EE. With thts preparation, 
'v. shall see how OR -provide, a metaphor for." ^* In the most useful and . 




powerful sense ot ntetaphor.. And finally, we shall consider ways In which 
wor^wrs may draw the two fields of activity, closer to each other. 



.. ,',The Corollaries of Deciston-MaWng' , 
Clearly, cvaluatlon»s claim to uniqueness,, to "being usefully l^eyond 
measurement, depends on Its *belng an aid to decision making. While not 
stated . explicitly. In much of the evaluations literature, this Ts at least 
Implfclt In much of. the writing vheit EE spokemcn attcopt to differentiate 

their field from the older one of measuremejit (for a review -of six sample 

— — t — 

texts, see Page, 1975). Evaluation is not the same as research, we 

are told,' because EE l»^ "concerned with the' Individual case," alms to 

"aid the planner, administrator, and practitioner,;* and Iqdced "aid In 

making decisions." For such purposes we "evaluate" 'programs, students, ptoposals, 

products, , and personnel " «il making, or at least 

influencing, decisions about thesS subjects of 'bur Inquiry. 

Having acknowledged this rf»latloni let us see whether we- dan deduct ' 

# • 

some corollaries of let* ' ,.• . ' — 

( ' 

" 1) Any decision mst necessarily concern Itself with choice among' 
alternatives. Ho proper "evaluation" "can be of Just one^ject. This point 
la curiously fussy In, most of the evaluations lltcfature, but.lt does 
occasionally appear. For example Scrlven (ch. 1 In Popham, 1974) t 

^ew If any usefyl evaluations avol'd the necessity to present . 
{Uta on the comparative performance of critically competitive • 
'products. All too often the. data refers to some pre-established 
Standards ot merit, and the- reader has no Idea whether one can 
do better- for less, or twice as well for 5 percent more, which is 
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the kind of Infopoaclon a. consumer wants, 
^ontlzute i.^r this, passaf 
features o£ our fIecisloa<^klngt - 



- ) 

Let us contlzute i this, passage^ for It will iLluolnate som 6thar^ 



• # #' ♦ It Is not too thrilling to d{scover chat an Injection 

of $100 f 000 worth of cooputer-asslsted Instruction (CM) can 

improve the math performance of a school by 15 percent it there 

Is a possibility that $lf5()0 worth of progranxoed texts woul'd do as 

well or bettex* There are few points where good evaluators 

distinguish themselves more clearly than in their choice of 

critical cofflpetltprs. [p* 13] 

One corollary of EE as decision-makings then» is the need for 9uch a list of 

i 

feasible alternatives * , 

2) A second feature ot any declslon\aaklng is sotaa value x^hich 

» 7" 

may serve, as a g^oaU In the above example it la to "Improve the math 
perforoance of a school*" It is not clear how any decisions may be 
intelligently made if the process 1* truly "goal-free»"* Indeed, it is 
'T)f tcn*requlred that such objectives be measured, as th^y are above, in 
a. ratio scale How *lse Is^one to Interpret an liaprovement of **15 percentf'? 
CleArly> then, to decide among altejnatlvesf we need such a prediction of 
outcome for each altematt^^ 

3) A third feature of decision making Is an estimate of costs , 

' ' ' . *^ 

accompanying each, alternative* In the Illustration abovei costs are 

_r • ■ . , " * 

»ln terms of *dollars ("$100,000 worth of computor-asslstod Instruction" vs. 

"^15,000 wor;th of prograiaped texts") v^If costs are the same* then wo choose 

terms of value; but 1£ values ace the same, then costs bocooe essential* 

But In the "roal world" .so prized by evaluators, some Information about" 



tloth posts and benefits is essential to maVln^g defensible decisions* * 

ifhfit Is "Still less acknowledged in EBf but is Implicit in the above 
<i^aizq>lei is the need to measure cos^s and benefits oh the same scale* 
Suppose we can improve a scbool*8 math performance "10 percent" by 
spending an additional "$^^000" — should w do so? Glearly^ we 
needy for mapy non*crivial questionSf, some way of plotting a function 

iof costs^ against b'enefits. <Tbi^ is a balancing ve perform all the 
time in real^ife: Else how would we decide to pay $3 f6r a movie?) 

•I 

\ . *4) And still a fourth crbrollary of ES as decision making is 



•ichisi We must have some knowledge of the probabilities of ^rioua 

outcomes for each alternative* I£ we choose A or choose that lS| 

w6 need to anticipate t^ "chances" of various resnltSf with theisr associated 
— » costs and benefits* 

If we know these certainly, It gxeatly simplifies our 'choice. But ^ 
in the much vaunted "real world" of Z&$ lotowledge about such probabili^ties 
must be wrung .from j^ature*s close reserve^ must be extracted like Juice' 
from any high-quality evidence we have* 

And here is It rather startllngt though unavoidable feature of these 
corollaries! '^These outcomes^ bo.th costs and benefits^ must be related 
^ causally to ^e decisions* Mere correj^atiozlal dat^^ collected froml 
artlfadtUal records either In this school or elsewhcrei will not suffice* 

I /• 

For no decision making is rational unless It^ b^ars on producing the 
more *desirabl^ Outcome, r * - V * go saying» ve have prepared 

^e way for models of Operations Research * Even moret We have made 
EE bfeavtly dependent of> traditional, knowledRO-almed researcbj for It ts^ 
from true research that we^an best hope tio understand the required 
cftusarl nexus * 



■JO 



■ 

» An O^^Model £or Decision Analysis .'^ * ^ 

Having fras^d our roquixemeats £ot d^ci$ipn making in evaluation^ 
ve nbw turn to a ccodel from 0»&« to show bow aH these requirements 
are confronxed a^d organized into ^a single strutotuFe^ Decision 

analysis (DA)^is only one o£ the toodels in 0R« But it seems the easiest ^ 

■ * ^ 

^0^ grasp., for intelligent educators unused to such thiillcing"^} and it serves 
as a kind ^£.*. , prototype .of the more specialised algor^thmsr^ 
described later in this article., ^^Ji is the only mode).^ ' 
shown here in stfcfficient detail. r \ / to be«': ,1 . . ' 
calculated. Its main features. are Illustrated Itt Figure 1* . 



Figure 1 , ^ 



Figure 1 represents a^sic^le decision tree containing Che following 
structures There Is a set of nodes and branches^ according to how 
tibe decision problem is envisioned by the designer. Th^re are Just two 
kinds o£ nodess recta atfilar no^es are decisions^ and the branches from 
a decision node ar» the alternatives among ^hicb we tmist choose* . Tbe^ 
circle nodes represent orobabllittes> or uncertainties as to outcomes* 
The branches, from a^ circle 4kre the posslbU events of interest* Each 
branch from a/circl'e node, wst be flagged by^ some estimated probability t 
ancl these prqbabilities must add up to one for any circle node« At the 
end of each terminal branch (whether from rectt^ngle or circle) must be 
^ome (black dot) value attached to that outcome^ or plan. To complete 
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. 7 

the notation for this alnple 'tree, we have tje little toll-gate, n^ar 
the letter B| whar6 we show the estimated cost of that branch* Costs 
and values, as wa hav^rnoterf, oust b.e transfonaad Into the sane unlts^ 

# • ■ 

of iDeasure, If both are necessary to Include In the problem. 

' Let us take such a tree as representing a trae picture of our 

'» ^ ^ 

decision probl^, and the prpbabllltles, costs, and values as being a • 
reasonable approximation to,, their true measures. Then to the 
•stonlshaet^t of newcomers to ~ such a tree Is solved entirely 

automatically, algorlthndcally, reaching decisions which are demonstrably 

< 

optimal. We begin at the bottom of th? tree with, the t^tmlnal values^.^ 

- ' * 

And we mdve up the tree by purely automatic procedures. ,If the node above 

Is a probability node, then we miltlpLy each value by Its probability, 
^ . / • '. . . . « . . 

thus eaiculatlng a mean value which becomes the value now attached to^ v 

Che Jiode Itself. For the probability (circle) nodes, these calculations • 

i ■ *' 

are shown In Figure !• ^ " " ^ ' 

, On the other. hand, where the node Is a decfslcm (squtfre)' node, wa 
"fold back" the branches of less value, Indicating thls^by the double ^ 
.barrier a^ seen nlar the letter A.^ A^id we. attach th^ bluest branch value 
to" the node Itself. We work up the tree this way,fro(a bottom to top, 
with these two* »lnple operations of averaging and folding back (and 
subtracting costs).. And In this simple, feQursive way, we can solve a 

, ' ^ % 

uee of any arbitrary^ complexity. *In this lllustratfpn of Figure 3, If , 
we*'were considering Plans A and B, we would autojuatlcally choose B, 
<lnce 70 Is greater than Wt ^ " 

The 'c'tarlous thln^ about this model about our neglec^ of It In ^ . 
education - is that .there Is no alternative theory of dcclfllon maklnfi . , 
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Xh9it itf there tre ^ther OR nuddeLst but there Is^ no rational way to * ' . 
decide eny^wftter but by consideration o£ the variables in this treei 
probabilities! coitSy and benefits* And ^iven thes^ elemaxits of the . 
prjbleiai then iucb ^ a oatheioaticaL^ sot^tioni as ve ^ave said> is ^ , ^-'^ 
demonstrably equal to or bettet *ttiai^ny subjective sdlutiira* (An 
iiaportant but nf gtected larea of res^archy by thc^ .vay. is the degree 
to vh^cb AS human deciders», can opttlmi:;e when ve^ stjoj^ectively 
solve treei of this kind* .But see tillattt 1973»} \ J * . , 

^ - > jAplf o£ Subjectivity in Oecision Analysis * ^ 

V * * > , " • ^ / ^ ^ ^ « 

* We must'notey in Figure ly^that there are ma^ aspects of the 
tree vherC commonly lack rii^n^datai o^tei^ tho^e Very ptoi)ab ill ties » 

<costS| ^d"" outcome values* T^en ve mu 9 1 ^depend on judgments ^ These 

'// \. ^ " ' ' ^ ^ ^ ^ 

Judgments^ consi^eij^d individually* are inevitably going to include large 

•^^ • • . * 

subjectiv^ coppotients* But such Inclusion .does not render the 

approach uoscientillt* to the contrai^* all science includes in tersub jec ttvt ty 

as a cardinal test oC iti s^tatus (Feigl & Brodbeck* 1953| ch. !)• As 

' / . ' ' . * ^ ^ 

Feigl points out» in ter sub jec tlvi ty is in fact another term^or 

, »# 

objectiviftyi the agreement at the most* elementary level about whether 

« tlnple event did or did .not occur* *An(>'one o£ tHe' comcaonest concerns 

In tocUl science le the "reEieblllty" o£ aewurest ,vhlcb In the caie of ^ ' 

more ctkspXex Judgmea^St such as those Mere cTSnsidered^ refers to the 

eoncoi^dance among Judges ai^ng in(&pendently* Thi^dlffitence bptveen^ 

' , ' - . • . > ' ' » ? 

ordixm^py subjectivity itnd ^£ I call judgment is the difference between 

unchecked, unverif iable. individual experie^de a kind of solipdism 
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and^e central, desirable inter subjectivity^ science* A virtue of 
^clsion science is that it tanes subjectivity* giving it lAatevcr 
role Is necessary, but «st\i!iating its quality* making sure it is » ' 
repreWtatlye, and usidg it carefully in algorithms vAi'ch ar? themselves 
objective* In a decision "tree such ad Figure 1, idiere does such ' 
Judgment enter? int^^). the desigiTof the tree,^ since there may bo _ 
m're than one stt of alternatives to conaideri 2) the estlpation *\ 

of costst, usina wjiwtever date are avallablef 3) ettlnatlon of pro'babiiities 

^ ' * > 

(though regression techniques may provide Helpful estloates)}. and * . 
«ntrally> inescapably, 4) the estimation of values* ' j. _ - 

V 2«tlmating Values foir Decisions ^ \* ^ ^\ 

• It is a curioul char^icterrsfic of theV'evaluatio^s" literature that 
it has paiid so- little systematic attention 'to values* .A great virtue 
of the Off approach is that it forces us to'reaedy this tack, and to 
make explicit some scale of worth. For A'-ppiae requiremeht pf OR ^ . 

is sop»-agreed^pon xab jecti^a function s Vaosturable valuer • «i^that* j 
un^ivocah^ rttlect the J^uture vell-l^^g of the organization"' (tfagnar?! 
1969>"P» ^^^A^Syy^^^^' ^"^^ **mea8urable valuest"" 

onco wa teaVcWP (deceptively) single measures such as dollars of .profit, 
or mlLfrs of travel saved* Vet ve rou^nely analyze ratings, rankings, 
grades*, and other judgoents in our literature, and there is no needed ^ 
eptstcmoloy;/ which is not already a i/ell»vom tool of behavioral science* 
Also, contrary to vha^ m^ny think, tfiere is* not an unbridgeable gulf 
between the bi^est philosophical values and the most discrete test items 
or behavioral objectives* At Connecticut, intrigued by .this problem, 

• / ' ' ' ^ ' . ' 

some of us have done VQtk on vfiat ve call the bantee for "benefit T-score*^* 



> • * ^ . • • * 10 

Ue hate explored what, we may tera a **top-down tree of value" (Pagey 19729 
1974; Fage^ Breen, I974a9 ll97Ab}* A sample tfee is showir in Figure 2., 



Figure 2 



With Figure 2^ ve can grasp the remarkablet recursive property of a 

■ 

v.aluetreey and explore some of Its mathfematlcal properties* Wer define 
the ben tee f at the topmost level 9 aa Including lOGfi of the 

value to be gained from an educational experience (In this lllustratlont 

• * 

the fuir elementary and seootedtfry experience 9 the bentee being for 

the senl6r graduating from high school)* We have divided this 

bentetf i Into 7 areas of educational Improvement (Verbal^ • ^ ^ * ^ 

* Quanta tatlvet etc«r shown}f and we have apportioned th^ values ~ 

according "bo the Judgments of 101 randomly selected Judges9 half 

laymen and half professional ed^cators^ The^sk appeers very easy for the 

Judge CO ddi We f Imply present bio/her with a V&t of these 7 traits, . 

together^ with a paragraph^ description of each* The^ are presented In a 

nay randdm order for each Judgei with a sheet of paper with the 7 areas 

narked, and a stack of lOO^chlps* The^ Judge ii asked to "spend"' the thlps* 

* * 

the JMthodfls eesy and qulck9 4nd thpugh Individual judges may be 

* ■ 
^vd^ldly devlant9 ^th'e group estlmatee rapidly reach stability as 

'sample slz'e^ Increases* ^ Wen ve have such estlmateS9 ve can then 

laake calculatAns wblch vere, formerly Impossible* For exaiiq>le>9 If ve 

have a group of hlgb-school se;rlors9 and standai^dl^ed measures or*' 

eatl|iate8 for them on each of the 7 tralt-s^ we may calculate for a 

^ student an overall bentee 9 by welghtlng^^^agh of ' 

V — 

t 



his 7 estimates spproi^riatelyv by sumning ichoso weighted estimates for 

hiSLf and by transformLno^is sum to a T^score (with a toean o£ 50 and 

'standard deviation o£ 10) # . Such bentee scores would now permit us to 

. - * \ • guidance!' , 

*rank order \ht students for 9 variety o£ purposes of selection/and award* 

The maatj^emitlcable €eaturt» o£ the value tretf is Its generitlity: 

Vejcah. use. the sam^xechniques at atiy node in the tree* We can define a new 

' bentee fqr^ ^ay^ Aiqertcan literature* collecting expert opinions, about 

the apportionment o£ value, building curricula and t^sts in accordance 

with these vslues^ and assigning overall scores as the weighted sum o£ 

-tests* Note, this is quite similar to ^diat is done in curriculum 

Aad test^onstruction todays but the vaXue-tree approach establishes^ 



a new legitimacy to the putcome^ since values are defined in a defensible way 

The.tree.haf additio'nal mathematical properties^ v^i^ permit , 

us. . .la. principle f y \- to weight any node within the overall scheme 

of values, by .-lArtfea adiipted - -zlxoei^a trte of indcpdnderit probabilities* 

(Cf. Bage^. 197Ai Pago.&^^recn,, 1974a*) And it may be" used. 

for vittijallyany purpose or sab-groupt third-gra'de pupils, learning* 

dl sable d:,tQen-%gers,'v . .?r. t , pre«med or engineering major's* 

*The txtt iV particuljarly useful Kbeit facing ^ the problem of t;radeqf f 1 

for ixampl^ where one gains in math at the expense of time taken 'from 
t * evaluation of * ^ 

vsocial studies^ sinc^it permits / people^ grwpSi inst£cutions> or prog^^ms 

in terms' of bverall educational ef f ecj:s^ 

-BQtir^ly different problems of valuefy of course^ ate raisediwfaen 
we attempt to trade ^f values against costs^when these are measure$i 
in different unitst foi^ example^ bentee points vs* hours of teacher time} 
or anytime we are faced with mor^ than one kind of scale* 



y ' ■ * ■ 

But such questions)' toO) have cooe under expejrt scrutiny by sotse ' 

Vorktrs In 0R« ^ Socu Issues v^re attacked at least 

two* decades ago Ce«g*'f Churchoany Acko££| *& Arnoff i t9S7y *c^« A)| 

and have been analyzed on a/ deep philosophical li^yel by t Chtnrchman 

(1961) » Scene « Important questions o£ .value^ functlbp. * xmre 

considered by Ral££a C19689 pp« 51-101} In his monograpli on OA. .And the most 

thor<iugh mathematical treatment / given value tradeo££s la probably 

the massive vork.by Keeney and Ral££a (1976) 1 which shCHjld proylde 

much stlmulatl^ and (.nsight £or behavloVal researchers • For our 

purposes herei va can ^ say that such problcuss are not Intractable^ scales 

can be drawn together 1>y Interview and experimental technlqucSi to 

establish the tradeb££s required. In order that decision trees wlll| 

Indeedi provide' a framework help£ul to the decision }makert , , 

- . . . / 

And the vdoit los^ortant realization o£ this secttoni £or those interested 

o£ value 

In EBt Is that these ccmplaxltleg/were not Invented by '0R > Rather, they 
reside In the real problems £aclng educatorsj and OR appears the only 
discipline prepared to make them explicit and to balance them o£^ * 
In an Intelligible way. ^ ^ , , , ' 

' • . * *^ * 

U * • ' - • . • ' 

Other Sub£lelds^o£ Operations Research « ^ « 

Suppose^ i^owf that^ are advising administrators on a different 

problemt the assignment of teachers to courses » Ve can asslg^^ch 

teacher, only jto a certain number of courseH^ and each required course 

in' the curriculum must be teOght* Ve have a specified goal,t Ue wish 

our assignment to optimize the, sum of ratings given by department 

^ • 3 

fibads to each teachef /course combination » How should we design • 
. V V set Wt to ■ . 

our OR model? We could /design a declsl97f tree, with every possible 

.17 



• * , If 

• : • . ■ 

teadter/courst -cocabi nation complex .having, a branch of its ovq* But 
. the number of such branches would become "ex treoely large* For example, 
1£ there vere Just 5 td&cbers and 20 courses for theo. ca cover (each. 

4 

teaching four bourses) ^ then there are 300 billion poselblo as^lgnmentftt 

fo;: fluch an asslgmoentiproblemy ' ^ 

EA Is obviously a poor wodtif theni and ve yrurn to the briaader field of * 
OR^for a more suitable choice # 

^ ^ Transportation Kodela 

For such problems^ a Suitable £raaevork Is an overarching ..^ne called 
a transportation tzu>del« Such a model provides 'one row for each 
Hgupply point" (In this casei r. a teacher) and one column for each 
''destination'' (In this casei a ^ourse)^ Instead of generating 
billions of possible asslgnsient tabloS| then^ ve are concerned with Just 
a 5 X. 20 table '(in this sample probUm)» and a limited number of 
^ Iterations over this table* Or If ther^*are| sayi only 7 unique courseSi 

then the tables Is enly 5 x 7 In thl% small example f .Such a reduced. table 
Is fllustrated lA''Flgure.3« 
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Figure 3 



As we note I for each teacher/ course combine tloni there Is a benefit r. .| 

5 , \ ^ . 

here*a rating of quality* For each cell In the Figure^ there will also 

* * * • - * It. 

%e tn X. .^wlth values ranging from zero (when the teacher teaches no sections 
of that course) to 4 (Iff sayi one teacher "teaches all four hours available 
In Course ' The aim of- the assignment algorltAm^ theni will -be to 

, maximize , - ^ 



5 7, 

This oaxlmlzatlcm Is carrled out under the Constraints »th)it. . 

r - - / ' 
•scb teacher Instxuct the muaber of agre/d hourSf and that each course 

section be covereft by mn instructor* 

An expert example of such anal^s .is provided by Tillett (1975)^' 

In a real-vorldy serious at^dy of such asaignzaent^y there are of course 

manyw consideraticrns beyon^ those abovet axl^ TitUtt adjusted his model 

for thesei For one thingf neither ratings nor preferences are alone 

enou^' to make out a good assignment* Both must be considered and 

balanced appropriately, sometimes in a way different for each teacher 

(dependilngf for example, on seniority)* This can be neatly done within 

the ^ame general algorlthic<p* 102)* • . 

^e / For anothert teacher preferences are \' complicated by . 

the number o£ sections assigned' of \m course (I may like to tehch 9th grade ^ 

f 

algebra, but four sections of it ?)* Tillett therefore 

extended, his model to a third^ dimei^sfon (the mis&ei: of sections of 
eacn ij combination) » And in order to make sure- he ob'tained feasible 

~ aolutlcnsi he abandoned the fast, algorithms for ene -called zero-one 
Integer pro^jcondng* Obtaliring data from 7 Connecticut high-school 
Math departoentSy he compared the optimal solutions for each, against 

^ ^e current schedule^ actually belilg taught* His results are shown 

- - * ' ; - • 

. ' « Table 2 ,' 



Haturally^^slnce the algorltlxm does optimize the objective 
function I* there is never a case where the solution .does not ^ 
toeet or surpass the current assigmoentf ^en only one dimension 
o£ benefit is considered* Thus; when preferences are maxlmlz^ed 
in Table 1$ -the preference gain is very clea^« * When we coc^are 
these oTAterence solutions ^ with the effectiveness ratings^ 
lioveverf we find there has been s6me tradeofff and the existing ' 
schedules are at times superior^ An'd we see the jfame ^^esultp. 
in reverse^ vh^n we maximize only the effectiveness ratings^ (Tillett 
did not apply tha available techniques for combining the two benefits 
in appropriate ways«) ^"^he clearest message from such workp thdn^ 
is that our choice of value dimensions is* of fundamental imipprtanco 

\ 

tf> our declsipn«matclng« . ' - 

Now suppose an evaluator is called'on t^ help make assignments 

;oi this general *sort* f Clearlyt the OS . framework can. enormous 1^^ 

Ai'd in knowlx^ what information to collect»^ vhat; oj;)inion8 an4 ^ 

preferencetf and how to cocaine them into a feasible and defensible 

solution^ There is also a side benefit frots nearly all models of 

operations researchi After finding one or more 6ptii^l'^olutions> 

one can do giensitivity analysis to find how stable it is under shifting 

estimatoji pf benefits And one can filrly easily calculate alternative 

soltttlonSf if the automatic ono rea^chbs produces some infoasible feature 

not anticipated in t)ie design* It helps us ask questions^ as well as 

malce use of th<b answers* ' . * 



\ Network Modeljs 



Another big ^ass o£ OR models can be adapted £or larftet t/nknown 

\ 



\ 

nusi>er o£ educational problems. , Theses are the network modelst 

which are probably most^Camlllar to educators as PEkX diagrams (e*g»t 

Qookf 1966) • But -there ate^'bountless other potential applications. 



For example t let us assume ^ question of ^rrlculum design* We have 
a certain amount o£ 'timet . a certain amount of material 

to cover (which we cannot.do lustlce to)t and eiiitlmates o£ tost and 
value £or different levels of effort for these . materials. 

How can OR help us sort out all this Infonoatlont and come up with the 
be$t possible design? 

For slmpllcltyt let usi'essume ^ we have just 10 hours of study^ 
tlme» . « to get through four chapters of a textbook*! For each* 
chapter we have a set of alternative levels of ef forty with a cost 
and value estimated for each level* If we jChlnk of beginning at one ^ 
end of a networkf going through the fcfir'^chapterst *and ending at the 
other end of the networkf we might draw the problem as In Figure 4* " 

. — * . / 



Figure 4 



, Now we mey -think of our problem as picking our way through 
a maze o£ pathS. He^e we are nQt juat picking the shortest pathf 
as we might In planning a trip) aifd we are not just picking the 
longest pathf 'ss we might when anticipating difficulties In PERT* 
Rather t we are picking that path Vhlch produces the mbst efficient 



txtp, g'alning us tl^ best ratio o£ bene£lt-to-cost, * ' 

po8sl|>Ie. within our set lioitation* ' , • < 

And/we .no|[ice a peculiarity of the network of Figure 4. There 
are five successive stages *of the trip, and there .is no line, for/^!ample 
directly collecting Chapter 1 with Chapter' 3.v In fact, this sort 
of problem is. therefore nicknamed the "stagec oach" problem (Wagner,* , ^ . 

/ * \ 

^19/91 p. 256} » tnd ir* permits Vch eastet^ solutions than i£ all 

/ 

lodes were connected with all lOtners* Because the.ioodel is' solved 

^by moving from one stage tcj tfee ne:^, solving each in turn, it 

•ia classified as a^Mynamic progresiaing' algorithm. 

Hecvork problems, lilce others in OR, allow, uj" to ^st. the ^ 

c . ' * 

aensitivity o£ the solution to errors in estimate^ and to try out 

various alternative values in the input. Here again, ve^ haVe a 

'frcmework which helps us ask the right queitions^ and, fit the answera 

into a rational set o£ suggestions^ together with soijto estimate: ot 

possible errors in .those 'jug^stions^ ^ • ^ ^ 

. , Siqulatioh Models 

In seeldng'Sttch legitimate approaches to evaluation, we have 
briefly considered three major kinds of. OR modelst .dccistorf' analysis, 
transportion, and network designs. These and others haVe in conmon 
the enornoua benefit of providing "optimal" solutions-, either maximizing 
or mlnlipizing so^e function ^jf agreed worth. But there is a large 
class of OR models, under thelgenerai head of "simulations", which . 
perodLt tis to se'arch ou£ a. large number of poss'ibillties, but which 
do not automatically home Irton the best alternatives. 



rEauldt;Lon modelst In £act| ^ire sowarlous as to be almost . ^ , 

Indescribable 9 s^nce they Include the vast quantity o£ OR. piroblenis 

< ' ' ♦ 

cannot be analyzed by one of the standard optimization 

modelsX Slnulatlon* Is very vhlua^lei Wo often vlsli to test a plan 

with^^ tb^s^omous .dangers and costs, of a real<*vorld trials so 

set out the vay'^tbe-^laxv Vould be expected to,work» often In 

\ also 
theSiorm of a flow-cbart* Ue/set out scene statistical features 

of the events we expect to encout^r^ often u&lrig random- 

number generators to simalate their occurrence* How we ^|t& a 

coa^uter pro-am (probably using some of the simulation j>ackages 

avalltible}| and run a large 'number of^cases^ tabulating the results 

In suifioary ways* ^We gain whatever wlsdotAwe may gather from^ 

'say^ /*expetiencing" lO^OOO' such'^ases* 

^ ^ Let^us consider a more concrete exaln^ltt*^ We ^d.sh classes lo make 

full tisel of the«.new' school If-brary being planned, but we ' are concerned 

about overwhelming the librarian ^d.th check-outs, possibly leading 

r ^ * ' 

to tardlnesSf confusion, poor disclplii^e*. Such overloads are qf&hdoa 

events^ but we b^e i^ome ^knowle^ge about the mian and variance 
■ cbtck«*ouC } ' * \ 

of /demand, for dassesT^oK^ there is a particular large brancS of ^ ' 
, slwlatlon, so wU derelopitd aAtf \ - that, it U iof ten treated 

separatelyi qutuej^nj^ theory* By ,the use of certain foroulai In this* 
Cbiory, we ari abl^\to **try out** several plans\£or the'libraryt fiinding 
Jor; each \r . how\ often we get Into ,dtf flculty^ and how bad the 
difficulty might bV. 

Llko the -other OR raodels, ^nulatton encourages u^to think hard 
' about out problaos, identifying <• key feature*, guiding our collection 



of Informatlont setting up our algorithms. When we run the problexoy 
ve get ^eaply what mLghc otherwise have eon very dearly* But whatever 
the nodeli the soluttona arif'onty as good as the match of the model 
vtth 'the world ^ 

V 

The OR Models As Meta'phors 
#thft arguiaent of th£6 article Is that dectslonApaklng Is the most 
Important distinction of educational evaluation) ana'^at Operations 
Research Is the science of declslon*makli^g# Thus» OR models would 
seem to be our preferred raetaphbr for evaluation* But the introduction 
of Ithls term should be justified* One scholar recently wrote that 
"ni systematic way for attacking metaphors exists at this time" (Guba^ 

! *■ ■ I ' 

I9j79)« X belieye« to .the contrary^ that some ways Qf attacking 

m^taphqr . (or.at least itff ?£irst cousin, analogy) have been brilliantly 

« 

syntheslxed In recent years* My preferred examples are all from the 

(e.g*»*Slagley 1971). 
new science of artificial Intelligence/^ For exampUt Thomas Evans 

(1968) has created an autoczktlc» rule-*drlven analyzer of geometric 

a^alogy^ problems of the sort sken In Figure ^5* 



1 

Figure 5 



His program ANMjOGY* even^decade' agot passed tests with the 
metaphorlc insight of an American ^ hlgh^achool student* 

In the more directly verbal realmt Reitman.Cl965) programmed 



a solution algorithm to the classic sort of word analogy: 
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. ' HAND J 'glove.:: FOOr^j <T0E,FINGER,SH(^E,LE(3), . ^ 

In chid * probleHi most vould^agree chat che chlrd cfaoicet 

c ■ * ■ 

SHO&t Is the correcc response* But vtirfi , Apmrenely because both ' 
sides o£ the e'quaclon will then saelsfy a relaelon o£ the formt * ' 

CLOTHING (HAND, GLOVE); and 
t • CU>THING<FOOT, SHOE); or taore* broadly, ' ^ 

where R Is a specified relationi and x and ^ are (ordered arguments 

i-Thlch satisfy that relation* ' ^ 

Such relations ere^at che heart of^iaefaphor analysis* . Indeed^ 

even geometrtc-^alogiea (as In Figure 5) are translated Into complex 

llstrpropesslng strings (using LISP) which express" many prppertlos . ' 
* ■ ■ and * ' ' * . 

of forms> /relations among chem* and permit pattern recognition by 

comparisons among such strings* Even visual metaphorsi tHenT 

6 

are converted and solved at che level of symbolic logic* 

Ixi eValuatloni - v we are seeking metaphors which will provide che 

^r - ^ . * ' . 

best springboard to /successful performance^ We seek ^comparisons of 

e ireal«*world problem (let us aayy Rxy) t^ich some other problem (^ay^ 

R'ab) ^ich will serve as a useful modal* Ue^ are apfjlyln^-^ out:> 

problem V^n Impriclt equation of che formx" 

^ - * 

Rxy « Rfab * 

there are two sources of dif f Icultyi 1} whether we have^ dlear idea of . 

che way che mdel^ itself workss R^abi and 2). whether t^ere Is indeed a 

_ * % - 

useful match of ch* two relations, diat Is* whether the nodel Is' usefully 

isomorphic irtch the reai<>vorld problem* 



On thcf first pointy OR seexos brilliantly ahead of any other n^taphor , 
of decldlon**mdking* OR has the enoraous advantage of all toachtiaaticst 
of bringing to bear a system of demonstrable theorems^ of certain truth* 
Vlrtaally no other^ non^mathematlcal loetaphor can hope to provide the ' 
Intellectual^nchor^ then, that OR provides* Compare Its working model 
of decision analysis^ for exas^le, with such intrinsically ambiguous 
metaphors as history or Jdutnalism* 

' On the second point « the fit of the model to the real*world problem » 
OR shares soma of the weaknesses of other metaphors* That Is, one must 
labor to eatabllsk that R really Is usefully equivalent to RS and that 
X and ^2 are usefully analogous to a. and b* of our OR 'model* As Herman ^ 
Campbell put It^ . ' ^ 

'It Is never. difficult to find a theory which will exj^Ialn the 
laws logjlcaliyt what is difficult is to find one \^lch will 
explain them logically and at the same time display the ^ > a 
^ requisite analogy* • * If it were found that the- analogy was 

false it would at qnce lose its Value* (Feigl & Brodbeck, l953| 

p. 289) * * - ■ ^ 

^ * • , . - 

.All metaphors^ then^ share the butden of proving their own -accuracy* 
But even here^ OR metaphors seem to enjoy unusual atatuSf sfnce they routinely 
proyide algorltbns for describing the real-world problem (for exadtple^^ 
by requiring listd of altemativesi and the values of probabilities, 

' costSi antt benefits) in wa^s which will at least pr6b6 ^or any falsitiy in the 

« ^ ^ * A ^ 

analogy* " * - . . * , 

When these two advantages are put together the power o^ the mathematical 



tooddl In Itsolfi and Its b^rtstl'c power In exploring the ^dimensions 
of fit to (he ^eal worlds- than seopis uniquely suited to^brovlde the 

mostruseful loediphors for educational ^valuatloni^vlch Its attendant 

^ ' *^ . ■ ' ' 

dyl8lo5-maIcingf . ' ^ ^ » ' 

^ ' ^ . ' '^OlMirations Rese^rth and(g^catlon 

This final section Is Intended as ^a ^Ide to ' ^ Che literature ' 
of 0R> especially of those topics most salient for educ^lonal 
e:\raluattont A masterful^ prtzewlnnlng siuznary of cha field Is. the 
lOX-page work by Wagner <1969}« This Is at once a hoge textbook and 
a ban(It»ook for professional workers In OR* There afe ve r y numerous 

exercises and word problems^ qt|lte a few with a behavlorsri or jaducatlonal 

^ ^ ' ' < 3- 

f layor« Other coo^rehenslve texts In OR are by Ullller ^nd Lleberman 

(1974)9 and by Gaver and Thompson <1973}* ^11 of these have many - 

problems^ and would be suitable for courses of two semesters or f&ore* 

For those seeking a particular esq>h8Sl^ on decision an^t^.slSf 



there Is an OR text by Trueman < t^74} » which devotes six chapters to 
such decision trees«. And If one seeks a cotablnatton of DA with our* 
iQore\ctasslc statistical apprpach* there are-ff«^at:lcs^ textbooks ^ 
Cvlthottt ^otker OR techniques) by rikdburg <1970) and b^lnkler and William L 
Hay^/(1975)9 .t^g^^ll^known psychologlcal^tatlstlcI3(r^ It seems certain 
^that nfere^ sta^ilrtics terxts.wlll move^ln the direction of DA* es Bayeslan 
approaches becocae .oore popular (^«g«f Novlck & Jackson^ 1974} » slnc6C«,J^ . 
there Is sone affinity between the two areas* And for .specialised 
introducclon to 'uKt th^ti It'stlLL no' botur work than that o£ Ralffa 
(1968)^ , • . 



23 



As ve.have notedfv psychologists an;! educators have been slow to 



discover OR9 but there are sotoe green shoots'^ o£ Interest # Some basic, 
OR concepts have .filtered into educatlout es^clally In* flow-charting 
, (e.g»> Kaufman, X972), EERTlrtg (Cook, X966), orJPPBS (Haftrtley, 1968). 
These bavl) typlca^y. used graphic aids flroza OR,* but sel<}|piir the 
advanced oathematlcal techniques,* let alone optimization, which are the 
core of OR as a professional discipline # In fact, there* wro at least 
two Introductory books for educators (Banghart, 1969; Van Dusaeldorp*,' — 
Richardson & F^ley,, 1971) $ but these are clearly at familiarization 

^ With aotne of fht concepts, and not at a level of useful expertise. 

. ' A specific OR textbook for educators and other behavioral scientists, 

^ then, seems a gap watting to be filled., • , . * 

Educators may not realize 'Just how advanced and conoM^jpUce OR is, 
in other fj^Ids than our*. , Courses exist In various university 

• * 

departmektd. Including statistics, engta'eetlng, industrial administration, 

y • ; ■ • . • 

,and sotnetiaes departoonts of their owti. Ph.D.*s are given In nany 

universities. In the U.S., large- and active professional groups are 

Idcused in major organizations such as the Ot)eration» Research Society ^ 

of American {ORSA) a^xd the Institute of Management Sciences (TIMS) . • 

These put out major «-*rhntr*l loumals. Ooeratlons Research and Hanaj^ement 

Science , and team up for the more applled;TIMS/ORSA Interfaces* and 

for large national and international -congresses, with programs running 

over 300 close pajes*. OR is at the heart of lauch serlpus; thinking 

in govemmont, industry,, and the military. And in Schools of Business 

Administration, the tj^oal MBA may routinely take two courses focused 

* _ 

on OR approaahes. Indeed, it seems to be that Education is the discipline 
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vfalcb Is *out o£ step In OR (or wblcb.^as hardly bcgon to xsarcht)* 

Yet there have been a nunber of efforts to Apply 0&. ^proacbes to 
the understanding of educational problems » and revlew^f some of these 
have been written tiy KcNamara (1971) and by Johnstone (4^74) • Kuch 
of the ;work^ha8 been done In higher education » since OR -specialists^ 
vlth the necessary abilities and skill S| have nost frccpxantly earned 
to nearby problems In thdlr own universities* - - 

It is clear that OR techikiques h'ave flourished iDost in fields 
^Aiit^ there are easily available objective functions, such as ttsie 
savedt or dollars i>rofltt or hits on target* Our educational objectives! 
hovever> are intrinsically less tractable; It^ls a remarkable achievement 
of the pasJt half century» that ve have (in Vefitem nations) reached 
some fragile consensus about the importance of sdoring veil on ^ 
obJe|^tive tests* And such laeasures ^ are an indispensable basis of 
good educationol' decision^making (for a teehnlcal, early approach, 
see Cronbach & Gleser, 1965)* 

Despite the remarkable, incremental progress of educational measurement 
there are- still problems *in using test scores as objective fjunctions for 
0R«^ One of tttese is that a test score, at best, is of interval^scale > 
measurement^ whereas for many OR applications we need ^l^tF^ strength of* 
ratio scale, that is, using a known zero t^oint* One solution is to . 
concentrate on thft change of t^st scores, since a zero point of a sort 
•is established for no-change* Yet there are psychometric prohJLcms,, 
even beyond the relative instability measurements of change* Here 
agoln» is « proolslng field for serious cesearcb. 
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Another pi^qbleta of »to5t scored, as we have seen, is putting 

in proper perspective as objectives of the edocatlonal eitterprlsetr 

Here we have, rfoted som strong basic wftk In inultlple values (Ralffa, 

^ ■ 
I966r Churchman, Ackoff & Amoff, 1956, ch. 6j Churchmn,rl961f and 

especially Keeney & Ra^f f a, I976y. More .applted work on this general 

^probiem- has been m ' dtted'CPage, 1972, 1973, l974j Pago & Breen,- 

l97Aa,t 1374b).' Th« bentte liiethods have also been appUed by Wayne 

HartttrCof the Kational Assessoent of Educational Progress) and by 

Dri William Stretch -/of the Faroington, Cona«, Public Schools)*. 

Ot^er major work on the, values of objectives has involved the 

»'bel^&J' techniqtfe (Dalkey, 1969), which has aimed .more at the creation 

r ' . ' ' ' ' ' 

of -consensus among a groop than at the discovery of independent" 

opinion. Hone of these have explicitly used values to cotabine test' 

scores into single measures (such ar described- in the bcntcc 

literature), but the -potentijl is clearly. present. 

.Apiat from test scores, of course, is the general use of tndopendent 

'racings of value^ on' a ratio scale. Once those are done, then OR 

techniques tnay be applied to optlmUe such values* And this has been 

' >j 
ddnc in curriculum design, ?s we have seen (Pago, Jarjoura & Konfljika, . 

\ ' ' 
1976} Page, 1976j Konopka, 1977J, in teacher assignment (Tillett, 1975), 

and most recently tn the curriculum planning and scheduling for 

Ji'ifth school students (Crandon, in press). In all of theje, however, 
* . " ' ' 

■the "applications" have consisted of the transfer of the general 

theory to a specific area, and the discovery of opinions, and the 

.(jesign and running of appropriate' programs to optimize the objectives. 



a^tvea the dat4. The 'applications h^4,n£^ involved the carrying ^ 




out of pja^s generated by OR. Such cooplcte applications rcaaln for 
the future* 

m 

** m 

WILL educational evaluation turn toward Operations Research » . 

evaluative 

as evaluators becooae more institutionalised, and the/ discipline takes 
on a more structured form? There is surely a movement lu this 
dtxectlony among at. least ^ few of the more quantitatively' |ophlstlcatcd 
^^^0 a^abiatlons , movement (e#g»| Edwards, Guttcntag & Snapper, t975j 
Comoy, 1976 a;»d Gene Glass In his Introduction of Camoy)# But 
the movement Is still very slight. And there is a strong counter- 
loovement, subjectlvlst and antl»technologlcal| also among major 
opinion leaders in evaluation (e*g*, Stake, 1978)* And we can 
p<y^t to other applied fields in education, (such as guidance) where 
there has been steady slide avay from the tedmical use of eyl^nce, 
more and more toward a **sof t" epi^tcmology, jdoctrine accepted 
falth,/^nd an ever heavier reliance on "authorities" in the field* 

In^^^Bucatlonal evaluation, we may have a rathl^ similar situation* 
We h^e a' strongly entrenched administrt^tlve lead»shlp, all having 

^ risen to their positions without any acquaintance with 0R# OR Is 

f ^ * ' 

little ttfifght In Departments of EducatlonaL Administration* When"" 
M% turn to research specialists, we have those who are, in Education, 
best prepared to understand the material* Yet there is still a 
major obstacle for them to overcome, as wall} for hardly any 
part of their training, neither analysis of variance nor multivariate 
analysis, will have prepared them to read the material easily* Which 
brings<us to thi single major problem, with OR and EE t It is hard* 
Even for measurement specialists, it requires months pi new effort 

• • • I. 31 ^ • 
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to grasp xhe'fundaoenAls. sFor the vaat nuahcr of si workers, who P ^ 



miinhe^r. of yt workers, who /( * 
sklll^ far oore^ than 



eophasize verbal, personal, and political iW.liy far nore^ than 
technical* ones, OR,a» « advencpd practicing specialty, may be • ' ^ 

hopelessly out'^of' reach* 



Ernest Anderson (1970)', recognizing the problem of OR*s Interface 

■ # 
vith education, as)ud about "The Little Man Who tsn*t Tbere'*t 

* 

♦ the Operations Researcher trained ^In an d^ dedicated 

to education? . ^ ' ' 

♦ ' the applied Educational Researcher who can and will £e«et 
V • out needed data and relationships even lofaen these hold 

/ no^roalse of a good Journal article or convention report? 

♦ the Client who can understand and use what The Operations 
and Educatiwal Researcher have to offer hln?. (p. 3) 

.We can repeat the*^que.8tlon. Yet "as C. P. €now wrote, '^The scientist 
has 'th'e future in his bones." Operations Research has' the most ^ 
to offer of any metaphor for the evaluative procese. It provides . 
a framework to tie together the beat Information and Insight Into 
a systea to help us raake Important decisions* In the long run, If' 
not Isinedlately, l^^taist prevail as the principal technology for 
evaluation* 
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- Footnotes 

!• Soc&e work bat been done in pushing^ the bentee at least a large 

step cl9Ser to ippllcation# Wayne Martin^ (of the National Assessmont of 

^- - I • - 1 

S<i(tcitlonal Progress) )ias explored values in Social Studies* Socoa of 

- * ^ , t 

us bave explored the redesign of a large and important coutse in electron's 

^th the U.^S« Navy (Page,& Oanfieldi 1975^ Page, 1976) « And Dr. Hilliaa 

Streicb Cof the Farmington^ Conn.^ Schools) has studied course structure 

in high school departitmnts of English and History* 



"""^^^.^.....X^ " the K*ll-kaown Bloom (et al^ 1956) Taxonbifay. of Edacattonal 
Objecttvis dldlnotTTraddreajt^tseJi to **values»» of the various goals^r ^k. 
in the way we are descrlbingr -^^t ^nd tbus^lsmot dlrectly^useful for choice 
among program* the various "levels" vithin the Taxonomy » on Ich^ o^ther 
band| can be weighted and incorporated vithin the bentee strd^eg^*^ 

3« could just: as well optlioize^ the sxtm of preferences of those 
toacherSf or the sum of ratings by studentSi or sooe cocnbinadon 'o£ any 
of these* 

4# This calculation treats-the courses as uniquei and ignores -/a ^tl n 
schedule constraints* Then the possible pemutations are .201/(41 41 41 41 41)* 

5* In the basic "transportation" ciodelf each cell might rather l>e 
flagged by a cost and the^^goil, would be to minimize the total cost 
of the final solution* Mathematically^ the ladjtcatment is trivial* 

6« The STUDENT programs of Caniet'^ B^obroif < 19687^15 perhaps "stlU c loser 
to the sort of metaphofr we seelc^ since it converts natural language (English) 
problems into algebra^ solves themi and transla^s the results back into 
English* ^ 

It 
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iUptlnUd from TilleCt (1975/ p* }93» .copyright Soeto->BconOtalfe Plaiihtng 
ScUncts^ Jjrtat Britain) t vttb the kind pennisslon of author 
and puWlfh«r# ^ 
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« 40+15+20 =s a 



JO(SO)+.70(tO) 
7+15=22 




Tlgur* !• A decl»lon tree * 



^ ' ^ ^Itdijoo b retched by 4racj>« out the hrtncbcs u f ^ u 

SEL^^riS! iiff^ 'r^ M-^nitL to <Sh bSSh iii", ?. 

node. P ooda are then toW^ (working from th« bottom up), by avtrairini out the h^nrLT 
A«d D ,»dB .re lolwd by^ ^oUIn, back oil but the most vi|'«We S*J^^,S,trf 
f^.vodtion,. the probability « ^ctetn,incd by ki^l^ ct S^ 
tod^ttjwe alio th« terminal value*. Ttehnical proedureTcan be a^lirf u>t3dt^ 
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Ptfture 2 » A Bentee 'tree of educational valuer If d^e bentee Is 
* defined at the **overall educatlorCal 'benefit' for a lZth«grada 
^ studeAtf then the values of each part of Interest In the tree nuiy 

^ ba daflned by tha token or other oathod* As noted by. this treeing 

Vout of poatlc'lntensts^ It Is possible to.move froa the highest 

philosophical value to the nost discrate ItAi or instructional 
objective 9 within only a fair subciivisiqniL or **generatlons«v ^ 
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Ch.2 



Figure A* A UxtboQk problem seen as network* ■ The student beslns 
«t the ,Ie£t atfd^eads et the right, having studied each chapter a't 
sooe level* Sach node has a cost, and a benefit* The best path from 

to B maximizes the benefit, vfaile keeping costs with soma permissible 

* ' i • 

amount* 
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Figar« 5. A "yon>8Crlc analogy" Inf IllRonc<-teat ttan > 
'of the tort tolvad th« coe^tar program AKALOQY 

(Evtntt 196{V^P• 271)* Tbt aia is to cotoplete'the rtlatlo* 
;*/?y^f to B as C is to vhatt^* by choosing corrtctly kmong 

tht fivt QUobtrtd tlternotl^ti* Tht accepted response 

is #4. Copyrisbt 19^8 by tha H^fl.T^ Prass, and 

. ' — ' ' / 

' rsp^intad wltl^ peraisaion of tha author and ^publishar* 



